Skip to content

feat: add built-in fetch skill (read webpage text) - #132

Open
hallerite wants to merge 1 commit into
mainfrom
feat/fetch-skill
Open

feat: add built-in fetch skill (read webpage text)#132
hallerite wants to merge 1 commit into
mainfrom
feat/fetch-skill

Conversation

@hallerite

@hallerite hallerite commented Aug 16, 2026

Copy link
Copy Markdown
Member

Adds a built-in fetch skill: pre-imported async fetch(url=...) that GETs a page with httpx (redirects followed, browser-ish UA) and returns its cleaned text — scripts/styles/templates stripped, tags collapsed, entities unescaped — truncated to max_chars (default 20k). Non-UTF-8 pages decode safely (errors="replace"). Network/HTTP errors come back as short Error: ... strings (data for the agent, not a crash).

Pairs with the search skill: search finds URLs, fetch reads them. A neutral prompt line advertises the skill and its signature when enabled.

  • src/rlm/skills/fetch.py — the skill
  • src/rlm/skills/__init__.py — registers fetch in the built-ins
  • src/rlm/prompt.pyFETCH_SKILL_PROMPT, emitted when the skill is enabled
  • README.md — documents fetch
  • tests: enable-stub + html_to_text markup stripping

Measured effect (paired A/B, redsearcher, 100 tasks/arm, same pin, skills-only diff, no sub-agents)

policy model regime WITH fetch WITHOUT delta wins/losses/ties sign-test p
deepseek-v4-flash uncapped 0.392 0.392 +0.000 14/14/69 (n=97) 1.0
Laguna-S-2.1 capped* 0.219 0.177 +0.042 17/13/66 (n=96) 0.59

*capped = per-session 100 turns, 100k context, 8k/call, 10k tool-out (identical across arms).

Reading: no significant reward gain from the skill itself. Mechanism from mining the control arms: a strong code model simply substitutes — deepseek hand-rolled 12,086 page reads (requests.get/urllib/curl) across 100/100 control rollouts, so fetch is a convenience wrapper for a capability the model already exercises. Laguna shows a small positive trend (+0.042, n.s.) with modest adoption (29/100 rollouts, 391 calls). The value of fetch is therefore primarily hygiene (clean text instead of raw HTML in context, uniform error strings, one sanctioned path that a truncation policy can target) rather than measured task reward.

Failure taxonomy from the runs (follow-up material)

2,971 fetch calls (deepseek arm): 33% error rate, dominated by 404 (835× — 631 on en.wikipedia.org: the model guesses /wiki/Title URLs from memory), then 403 (71), 429 (34). 28% of successful reads hit the 20k max_chars cap. Suggested follow-ups:

  1. On 404, append a hint ("page not found — use search() to find the real URL") or auto-fallback to the site's search endpoint for known hosts (Wikipedia opensearch).
  2. Continuation support (fetch(url, start_char=N) or a trailing "N more chars available" note) so truncated reads can page through.
  3. Respect Retry-After on 429.

Also planned per review: expose fetch as a tool as well as a skill.

First of a 4-PR stack: fetch (this) → contract-only config (#160) → execution guardrails (#158) → role-aware sub-agent prompting (#151).


Note

Medium Risk
Enables outbound HTTP from the IPython kernel when the skill is turned on, which expands attack surface and context size (large page reads), though behavior is bounded by truncation and matches existing network-capable skills like bash/requests.

Overview
Adds an opt-in fetch built-in skill via RLM_SKILLS, alongside edit and search, so agents can call await fetch(url="...") from IPython instead of ad-hoc HTTP/curl.

The new rlm.skills.fetch module GETs pages with httpx (redirects, timeout, default UA), normalizes scheme-less URLs, converts HTML to plain text (strip script/style blocks and tags, unescape entities), truncates at 20k chars by default, and returns short error strings on failure rather than raising. Registration writes the usual session stub like other kernel skills; the system prompt gains FETCH_SKILL_PROMPT when fetch is enabled. README documents the skill and RLM_SKILLS value. Tests cover stub generation and html_to_text markup stripping.

Reviewed by Cursor Bugbot for commit 9c5594d. Bugbot is set up for automated code reviews on this repo. Configure here.

Comment thread src/rlm/prompt.py Outdated
@hallerite hallerite changed the title feat: add built-in fetch skill (read web page text) feat: add built-in fetch skill (read webpage text) Aug 28, 2026
Comment thread src/rlm/prompt.py
"several angles at once, fan out with `asyncio.gather(search(...), search(...))`."
)

FETCH_SKILL_PROMPT = (

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is this prompt here? other skills are given with a description in a generic way, i don't understand why this is so privileged. not a fan of such special cases unless there's a strong need.

mikasenghaas
mikasenghaas previously approved these changes Aug 28, 2026
Pre-imported async `fetch(url=...)` skill: httpx GET -> cleaned text (scripts/styles/tags
stripped), UTF-8-safe decode, truncated to max_chars. A neutral prompt line advertises the
skill and its signature when enabled, mirroring the existing edit/search skill lines.
Pairs with `search` (search finds URLs, fetch reads them).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants